REAPER – GitHub Secret Harvester

REAPER is a continuous, high‑performance scanner written in Go that hunts for exposed secrets across all public GitHub repositories. It scans code files, pull requests, issues, and commit messages for API keys, passwords, tokens, emails, database connection strings, security advisories, and other sensitive data.

Features

• Continuous operation – Runs forever, sleeping between configurable cycles.
• Advisory scanning – Optionally checks repositories for known GitHub Security Advisories (CVEs).
• Multi‑threaded – Configurable worker pool (default 20) for concurrent scanning.
• Comprehensive secret patterns – AWS keys, GitHub tokens, Slack/Discord/Stripe keys, JWT, database URLs, private keys, emails, generic passwords and API keys (with entropy filtering).
• Smart email filtering – Automatically hides obfuscated emails (user[at]example[dot]com) and GitHub no‑reply addresses. Deduplicates emails per repository.
• Rate‑limit aware – Respects GitHub API limits; backs off when limits are hit.
• Stateful scanning – Remembers already scanned repositories (scanned_repos.txt) to avoid re‑scanning.
• Real‑time output – Findings saved to both CSV (masked) and JSONL (unmasked) files.
• Single or batch mode – Scan one repository, a list from a file, or continuously scan everything.
• Graceful shutdown – Saves state on Ctrl+C.

Prerequisites

• Go 1.21+ (only needed for building from source)
• GitHub personal access token (classic) with the repo and public_repo scopes.
  Create one at: https://github.com/settings/tokens
  For advisory scanning (-scan-advisories=true), also need security_events scope.

Installation

Clone the repository and build the binary:

git clone https://github.com/ekomsSavior/REAPER.git
cd REAPER
go mod tidy
go build -o reaper reaper.go

Usage

Set your GitHub token as an environment variable:

export GITHUB_TOKEN=ghp_your_token_here

Command‑line flags

Flag	Default	Description
-workers	20	Number of concurrent repository scanners
-verbose	false	Print each finding to the terminal
-continuous	true	Run forever, sleeping between cycles
-sleep-minutes	60	Minutes to sleep between continuous cycles
-since-days	7	Only scan repos updated in the last N days (0 = all)
-scan-prs	true	Scan pull request titles, bodies, and comments
-scan-issues	true	Scan issue titles, bodies, and comments
-scan-commits	true	Scan commit messages
-scan-advisories	false	Check each repository for GitHub Security Advisories
-hide-obfuscated	true	Hide obfuscated emails (e.g., user[at]example[dot]com)
-entropy	true	Enable entropy checking (reduces false positives for API keys)
-min-stars	0	Minimum number of stars a repository must have
-output	./output	Directory where output files are saved
-repo	""	Single repository URL to scan (can be used multiple times)
-repo-list	""	File containing repository URLs (one per line)

Modes of operation

1. Continuous mode (scan all public repositories)

./reaper -workers=20 -verbose -continuous -sleep-minutes=5 -since-days=0

2. Single repository scan

./reaper -repo https://github.com/owner/repo -verbose

3. Multiple repositories (using multiple -repo flags)

./reaper -repo https://github.com/owner/repo1 -repo https://github.com/owner/repo2 -verbose

4. Batch scan from a file

Create a file repos.txt with one URL per line:

https://github.com/owner/repo1
https://github.com/owner/repo2

Then run:

./reaper -repo-list repos.txt -verbose -scan-advisories=true

5. Quiet mode (minimal output)

./reaper -workers=30 -sleep-minutes=5 -since-days=0

Only dots (.) are printed for progress; findings are still saved to files.

Output Files

All findings are saved inside the output/ directory (or a custom path set by -output).

• reaper_findings_TIMESTAMP.csv – CSV file with masked secret values (safe to share).
• reaper_findings.jsonl – Newline‑delimited JSON file containing unmasked secrets. Handle this file with extreme care.
• advisories.jsonl – Newline‑delimited JSON file containing security advisories (when -scan-advisories=true).
• scanned_repos.txt – List of already scanned repositories (full names). Prevents re‑scanning across cycles.

Secret Patterns Detected

REAPER currently looks for the following patterns (non‑exhaustive list):

Category	Examples
Cloud Keys	AWS Access Key (AKIA...), AWS Secret Key, Google API Key (AIza...)
Tokens	GitHub Token (ghp_..., github_pat_...), Slack Token (xoxb-...), Discord Bot Token, Stripe Live Secret Key (sk_live_...), JWT
Databases	PostgreSQL, MySQL, MongoDB, Redis connection strings
Private Keys	RSA, SSH, EC private keys
Credentials	Email addresses, Generic passwords, Generic API keys
Other	Azure connection strings, Twilio, SendGrid, Heroku, OpenAI, Telegram, GitLab, Docker Hub, NPM, Pulumi, DigitalOcean, Alibaba Cloud

Email Filtering Logic

REAPER intelligently filters email addresses to reduce noise:

1. Obfuscated emails – Patterns like user[at]example[dot]com, user AT example DOT com, user@[domain] are excluded (unless -hide-obfuscated=false).
2. GitHub no‑reply – noreply.github.com and users.noreply.github.com addresses are excluded.
3. Deduplication – Each email appears only once per repository, even if found in multiple commits or files.

Continuous Operation Details

When -continuous=true (the default), REAPER performs the following loop:

1. Searches GitHub for public repositories using the query a is:public (optionally filtered by pushed:>date).
2. For each new repository (not in scanned_repos.txt), it scans:
3. All files in the default branch.
4. All pull requests (titles, bodies, comments).
5. All issues (titles, bodies, comments).
6. All commit messages.
7. Security advisories (if enabled).
8. Findings are written to CSV and JSONL in real time.
9. After the current search result pages are exhausted, REAPER sleeps for -sleep-minutes.
10. The cycle repeats, picking up newly created or updated repositories.

Press Ctrl+C at any time to stop gracefully; the list of scanned repositories is saved.

Detailed jq Walkthrough: Analyzing REAPER Findings

The reaper_findings.jsonl file contains unmasked secrets in JSON format, one object per line. Here is how to use jq to analyse the data.

Basic viewing using jq

# View all findings (formatted)
jq '.' output/reaper_findings.jsonl

# View raw lines (useful for counting)
cat output/reaper_findings.jsonl | jq -c '.'

Filtering by secret type

# AWS keys only
jq 'select(.SecretType == "AWS Access Key" or .SecretType == "AWS Secret Key")' output/reaper_findings.jsonl

# Email addresses only
jq 'select(.SecretType == "Email Address")' output/reaper_findings.jsonl

# GitHub tokens only
jq 'select(.SecretType == "GitHub Token")' output/reaper_findings.jsonl

# JWT tokens only
jq 'select(.SecretType == "JWT Token")' output/reaper_findings.jsonl

Filtering by severity

# Critical severity only
jq 'select(.Severity == "CRITICAL")' output/reaper_findings.jsonl

# High severity only
jq 'select(.Severity == "HIGH")' output/reaper_findings.jsonl

# Critical or High
jq 'select(.Severity == "CRITICAL" or .Severity == "HIGH")' output/reaper_findings.jsonl

Filtering by repository

# Findings from a specific repository
jq 'select(.Repository == "owner/repo-name")' output/reaper_findings.jsonl

# Findings from multiple repositories
jq 'select(.Repository | test("owner1|owner2"))' output/reaper_findings.jsonl

Filtering by file location

# Secrets found in actual code files (not issues/PRs/commits)
jq 'select(.FilePath != "issue" and .FilePath != "pull_request" and .FilePath != "commit_message")' output/reaper_findings.jsonl

# Secrets found in environment files
jq 'select(.FilePath | endswith(".env"))' output/reaper_findings.jsonl

# Secrets found in configuration files
jq 'select(.FilePath | test("\\.(yaml|yml|json|toml|ini)$"))' output/reaper_findings.jsonl

Statistics and counting

# Count findings by type
jq -r '.SecretType' output/reaper_findings.jsonl | sort | uniq -c | sort -rn

# Count findings by severity
jq -r '.Severity' output/reaper_findings.jsonl | sort | uniq -c | sort -rn

# Count findings per repository
jq -r '.Repository' output/reaper_findings.jsonl | sort | uniq -c | sort -rn | head -20

# Total number of findings
jq -s 'length' output/reaper_findings.jsonl

Extracting specific fields

# Show only repository, file path, secret type, and secret value (tab-separated)
jq -r '[.Repository, .FilePath, .SecretType, .SecretValue] | @tsv' output/reaper_findings.jsonl

# Show only critical findings with repository and URL
jq -r 'select(.Severity == "CRITICAL") | [.Repository, .SecretType, .URL] | @tsv' output/reaper_findings.jsonl

# Export all findings to CSV (unmasked - be careful)
jq -r '[.Timestamp, .Repository, .FilePath, .LineNumber, .SecretType, .SecretValue, .URL, .Severity] | @csv' output/reaper_findings.jsonl > all_secrets.csv

Finding unique values

# List all unique email addresses found
jq -r 'select(.SecretType == "Email Address") | .SecretValue' output/reaper_findings.jsonl | sort -u

# List all unique repositories that leaked secrets
jq -r '.Repository' output/reaper_findings.jsonl | sort -u

# List all unique API keys (excluding emails)
jq -r 'select(.SecretType != "Email Address") | .SecretValue' output/reaper_findings.jsonl | sort -u

Time-based analysis

# Findings from the last hour (requires adjusting date format)
jq 'select(.Timestamp > "2026-04-14T12:00:00Z")' output/reaper_findings.jsonl

# Show findings with timestamps
jq -r '[.Timestamp, .Repository, .SecretType] | @tsv' output/reaper_findings.jsonl

Analysing advisories (if enabled)

# View all advisories
jq '.' output/advisories.jsonl

# Critical severity advisories only
jq 'select(.Severity == "CRITICAL")' output/advisories.jsonl

# List all vulnerable repositories
jq -r '.Repository' output/advisories.jsonl | sort -u

# Count advisories by severity
jq -r '.Severity' output/advisories.jsonl | sort | uniq -c

# Find advisories for a specific repository
jq 'select(.Repository == "owner/repo-name")' output/advisories.jsonl

Combining with grep for context

# Find secrets containing a specific keyword (e.g., "prod", "live")
jq -r 'select(.SecretValue | test("live|prod", "i")) | [.Repository, .SecretType, .SecretValue] | @tsv' output/reaper_findings.jsonl

# Find secrets that look like they might be valid (long entropy)
jq 'select(.SecretType != "Email Address" and (.SecretValue | length > 20))' output/reaper_findings.jsonl

Real-time monitoring

# Watch new findings as they arrive (like tail -f)
tail -f output/reaper_findings.jsonl | jq '.'

# Watch only critical findings in real time
tail -f output/reaper_findings.jsonl | jq 'select(.Severity == "CRITICAL")'

# Watch only email findings (non-obfuscated)
tail -f output/reaper_findings.jsonl | jq 'select(.SecretType == "Email Address")'

Exporting for reports

# Generate a summary report
jq -s '
  "REAPER FINDINGS REPORT\n" +
  "=====================\n\n" +
  "Total findings: \(length)\n\n" +
  "By severity:\n" +
  (group_by(.Severity) | map("  \(.[0].Severity): \(length)") | join("\n")) +
  "\n\nBy type:\n" +
  (group_by(.SecretType) | map("  \(.[0].SecretType): \(length)") | join("\n"))
' output/reaper_findings.jsonl

# Generate HTML report (simple)
jq -r '
  "<html><body><h1>REAPER Findings</h1><table border=1><tr><th>Timestamp</th><th>Repository</th><th>Type</th><th>Severity</th><th>File</th><th>URL</th></tr>" +
  (.[] | "<tr><td>\(.Timestamp)</td><td>\(.Repository)</td><td>\(.SecretType)</td><td>\(.Severity)</td><td>\(.FilePath)</td><td><a href=\"\(.URL)\">link</a></td></tr>") +
  "</table></body></html>"
' output/reaper_findings.jsonl > report.html

Troubleshooting

"No new repositories found"

• Your GitHub token may be invalid or missing the required scopes.
• The search query a is:public may be rate‑limited. Wait a minute and try again.
• If you used -since-days=0, there are always repositories – check your network connection.

High rate limit hits

• Reduce the number of workers (-workers=10).
• Increase -sleep-minutes (e.g., 30 or 60).
• Use a GitHub token with higher rate limits (authenticated requests allow 5,000 per hour).

False positives (e.g., markdown links flagged as passwords)

The "Generic Password" pattern is intentionally broad. You can:
- Disable entropy checking for passwords (-entropy=false) or adjust the regex in GetAllPatterns().
- Post‑filter the JSONL output using jq.

Obfuscated emails appearing

By default, -hide-obfuscated=true. If you still see obfuscated emails, add more patterns to the isObfuscatedEmail() function in the source.

Building and Testing Locally

1. Clone the repository.
2. Run go mod tidy to fetch dependencies.
3. Build with go build -o reaper reaper.go.
4. Test with a single repository:

bash ./reaper -repo https://github.com/ekomsSavior/REAPER -verbose

Disclaimer

REAPER is intended for educational purposes and authorised security assessments only.

MIT License Copyright (c) 2026 ek0mssavi0r / Church of Malware Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. USE AT YOUR OWN RISK. NO WARRANTY PROVIDED.